Determining tap locations on a handheld electronic device based on inertial measurements

ABSTRACT

Systems and methods are described in which the location of a tap on the body of a handheld device is determined in real time using data streams from an embedded inertial measurement unit (IMU). Taps may be generated by striking the handheld device with an object (e.g., a finger), or by moving the handheld device in a manner that causes it to strike another object. IMU accelerometer, gyroscopic and/or orientation (relative to the magnetic and/or gravitational pull of the earth) measurements are examined for signatures that distinguish a tap at a location on the body of the device compared with signal characteristics produced by taps at other locations. Neural network and/or numerical methods may be used to perform such classifications. Tap locations, tap timing and tap attributes such as the magnitude of applied forces, device orientation, and the amplitude and directions of motions during and following a tap, may be used to control or modulate responses within the handheld device and/or actions within connected devices.

RELATED APPLICATION DATA

The present application is a continuation of co-pending application Ser.No. 17/874,253, filed Jul. 26, 2022, the entire disclosure of which isexpressly incorporated by reference herein.

TECHNICAL FIELD

The present application relates generally to systems and methods for anindividual to perform machine-based interactions using a handheldelectronic device. Although the handheld device may be used by anyone,it is particularly well-suited for use by a young child, utilizingsimple interactive movements that lack requirements for precision manualdexterity and/or understanding complex interactive sequences. Systemsand methods herein employ techniques within the fields of computerprogramming, electronic design, firmware design, inertial measurementunits (IMUs), ergonometric construction, device controls, human motorcontrol and human-machine interactions. Systems and methods may providea user, especially a young child, with an intuitive machine interface torapidly and/or instinctively interact within an environment composed ofreal and/or virtual objects.

BACKGROUND

In recent years, the world has become increasingly reliant on portableelectronic devices that have become more powerful, sophisticated anduseful to a wide range of users. However, although children may rapidlyembrace using some aspects of electronics designed for more experiencedusers, young children may benefit from having access to interactiveelectronic devices that are small, light-weight, colorful, playful,informative, ergonomically designed for a child (including beingchild-safe), and easy to use. The systems and methods disclosed hereinmake use of recent advances in the fields of haptic technologies, soundgeneration using miniature speakers, portable displays and inertialmeasurement units (sometimes also referred to as inertial motion units).

Within a handheld device, alerts may be generated by a haptic unit (alsoknown as kinaesthetic communication) and/or a miniature speaker. Hapticunits generally employ an eccentric (i.e., unbalanced) rotating mass orpiezoelectric actuator to produce vibrations that can be felt. Alongsimilar lines, the vibrations of a miniature speaker are generallyproduced using a traditional (i.e., associated with larger speakers)electromagnetic moving coil or piezoelectric (so-called buzzer) designs.

Two-dimensional visual displays are composed of any number ofmonochromatic or multi-colored, addressable light-sources or pixels.Displays may range from a single light source (e.g., illuminating anorb, transmitted via a waveguide), to those that are capable ofdisplaying a single number (e.g., seven-segment display) or alphanumericcharacter (e.g., a five-pixel by eight-pixel array), to high-resolutionscreens with tens of millions of pixels. Regardless of scale, displaysare typically implemented as: 1) a two-dimensional array of lightsources (most frequently light-emitting diodes (LEDs), or 2) two platesof polarized glass that sandwich liquid crystal material (i.e., forminga liquid crystal display or LCD) that responds to an electric current byallowing different wavelengths of light from one or more illuminationsources (i.e., a backlight) to pass.

Inertial measurement units (IMUs) may incorporate any or allcombinations of: 1) linear accelerometers measuring forces generatedduring movement (i.e., governed by Newton's second law of motion) in upto three axes or dimensions, 2) gyroscope-based sensing of rotationalrates or velocities in up to three rotational axes, 3) magnetometersmeasuring magnetic field (i.e., magnetic dipole moment) including fieldsgenerated by the earth, and/or 4) the gravitational pull of the earth(including gravitational orientation) by measuring forces on an internalmass. The accuracy of IMUs varies widely, depending on size, operatingrange, compensating hardware that may be used for correction ofmeasurements (affecting cost), environmental factors including thermalgradients, the availability of individual device calibrations, and(integration) time required to perform measurements.

Advances in both electronics (i.e., hardware), standardizedcommunications protocols and allocation of dedicated frequencies withinthe electromagnetic spectrum have led to the development of a wide arrayof portable devices with abilities to wirelessly communicate with other,nearby devices as well as large-scale communications systems includingthe World Wide Web. Considerations for which protocols (or combinationsof available protocols) to employ within such portable devices includepower consumption, communication range (e.g., from a few centimeters tohundreds of meters and beyond), and available bandwidth.

Currently, Wi-Fi (e.g., based on the IEEE 802.11 family of standards)and Bluetooth (managed by the Bluetooth Special Interest Group) are usedwithin many portable devices. Less common and/or older communicationsprotocols within portable devices in household settings include Zigbee,Zwave, IR (infrared), and cellular- or mobile phone-based networks. Ingeneral (i.e., with many exceptions, particularly considering newerstandards), compared with Bluetooth, Wi-fi offers a greater range,greater bandwidth and a more direct pathway to the internet. On theother hand, Bluetooth offers lower power, a shorter operational range(that may be advantageous in some cases), and less complex circuitry tosupport communications.

Advances in miniaturization, reduced power consumption and increasedsophistication of electronics, including those applied to displays, IMUsand telecommunications have revolutionized the mobile device industry.Such portable devices have become increasingly sophisticated, allowingusers to concurrently communicate, geolocate, monitor exercise, trackhealth, be warned of hazards, capture videos, perform financialtransactions, and so on. Systems and methods that facilitate simple andintuitive interactions with a handheld device, particularly for use bychildren, may be useful.

SUMMARY

In view of the foregoing, systems and methods are provided herein thatdescribe a light-weight, simple-to-use and intuitive handheld devicethat may be particularly well-suited for machine-based interactions by ayoung child. Although the device may, in part, be accepted by a child asa toy, the computational flexibility embedded within the device mayallow the device to be used as a means for both machine-based and humaninteraction (particularly involving individuals who may be locatedremotely), play, embodied learning, emotional support, communications,expressing creativity, and enhancing imagination. Additionally, aportable, “fun” handheld device may motivate physical movement by achild (or adult) including kinetic motions and kinesthetic activities.

Young children may observe older children and adults perform a widerange of activities over extended periods of time using mobile devices,such as cell phones and tablets. However, young children generally needto first develop precision motor skills (e.g., to touch a specific iconon a touch-sensitive display) and intellectual sophistication tonavigate graphical user interfaces (GUIs) on such devices to achievedesired ends. Even the notion of pressing on one or more simplepushbuttons (e.g., that might be painted in bright colors) to achieve adesired goal is generally a concept that must first be learned by ayoung child. However, tapping anywhere upon a handheld, lightweightdevice body, or tapping the device against a solid surface, particularlyin response to a visual, tactile or auditory cue, may provide a childwith an intuitive method to use a handheld device within mixed, realand/or virtual environments without substantial instruction or precisionmotor skills.

Determining both the presence (including timing) of a tap as well as itslocation on the handheld device (including any affixed components) maybe determined by classifying electronic signatures from data gatheredfrom at least one inertial measurement unit (IMU) embedded within thehandheld device. As outlined in the Background section above, IMU datamay be derived from sensing combinations of translational acceleration,gyroscopic (i.e., rotational) velocity, magnetic force (i.e., includingforces applied by the earth's magnetic field) and/or gravitational force(i.e., resulting from the earth's large mass). In most multi-dimensional(i.e., up to three axes) implementations, IMU data may be consideredtime-varying sequences of vectors, containing both magnitude anddirectional information of sensed accelerations, orientations and/orforces.

A finger (i.e., of either hand) is a convenient and intuitive “tool” totap a handheld device. Within the English language, the term “finger”may convey an ambiguous meaning. Within much of the scientific world,“finger” refers to any appendage of a hand used for manipulation andsensing. However, in some instances (including in medicine), the thumb(containing two bones or phalanges) is considered distinct from fingers(containing three phalanges) due to differences in size, rigidstructures, joints and/or function. Within descriptions herein, the term“finger” or (interchangeably) “digit” refers to any hand protrusion,including the thumb.

Along similar lines, the verb “tap” is used herein to describe an actioninvolving the striking of one object against another object with adiscernible moment of contact, sudden change in acceleration, change indirection, and/or other discernible signal(s) that contact betweenobjects has been made. Once contact is made, either object may beretracted from the striking process, a sound may be produced, and/oreither object may temporarily be deformed during the process. As a nounwithin the English language, tap has a number of meanings includingreferring to the sound sometimes made upon striking objects together. Asused herein, “tap” refers to the overall action or process of strikingtwo objects together. Also, as noted below, the singular forms “a,”“an,” and “the” include plural referents unless the context clearlydictates otherwise.

Tap forces may be transmitted to the one or more embedded IMUs as aresult of a tap on the handheld device body including any affixedcomponents (i.e., forming a solid structure), where such rigidstructures transfer forces in the direction of the tap during the timeof impact (i.e., during the tap). In accordance with Newton's secondlaw, such forces may result in translational movements of the handhelddevice in any of three dimensions (i.e., often denoted as X, Y and Z,see FIG. 5 ). Forces may also result in rotational movements of thedevice (i.e., often described as pitch, roll and yaw, see FIG. 5 ) wherethe geometric center of the device and/or the functional center of anIMU may, for example, be used as a reference (i.e., an origin) as onemethod to describe such motions. During mathematical analyses, it mayalso be convenient to consider other locations (e.g., contact points ofa hand holding the device, contact point of a tap) to position theorigins of coordinate systems and/or vectors representing forces,orientations (e.g., relative to the gravitational pull or attraction ofthe earth), accelerations, velocities and motions.

A tap may be generated by a user by tapping a finger of an opposing hand(i.e., the hand opposite the hand holding or cradling the device) ontothe device, tapping an opposing hand (e.g., knuckle, palm) onto thehandheld device, tapping a digit of the hand holding the device onto thedevice, tapping any other body part of the user onto the handhelddevice, using a solid object such as a pen or stylus to tap on thedevice, or tapping the handheld device itself onto an object or surface(i.e., including a body part such as a knee or wrist; or other objectsuch as an image or text within a page of a book, desktop, floor, and soon). A tap may also be generated by tapping the handheld device againstanother handheld device (e.g., while mimicking battle using swords orsabers). Resultant actions to taps between devices may, for example, bespecifically restricted (i.e., by sensing tap location) to tapping aselected display on one device onto a target display of the secondhandheld device (e.g., during a process of exchanging informationbetween devices and/or device users).

When causing a location on the handheld device to impact a surface,there is no requirement for the surface to be absolutely rigid. Forexample, when tapping the device onto one's knee or the palm of anopposing hand, there may be some temporary deformation, at least at thelevel of the skin, during the tap. As further examples, when tappingstuffed or stretchable toys, there may be some flexibility of surfacesbeing tapped. Along similar lines, when tapping on the page of a book ormagazine, there may be some give or reactionary movements by the pagebeing tapped.

As described in greater detail in the Detailed Description below, IMUdata may be subject to one or more classification processes not only todiscern tap location on the handheld device, but also to discern othercharacteristics of the tap, collectively referred to as tap“attributes”. Tap attributes include measures of mechanicalcharacteristics of the object used to tap, magnitudes and directions offorces applied during the tap, movements (including magnitudes anddirections) of the handheld device immediately after a tap, theorientation of the handheld device during a tap relative to either orboth of the magnetic and gravitational pull or attraction of the earth,and so on.

Such differences in sensed accelerations and forces that are under thecontrol of the device user may allow types and/or attributes of a tap tobe classified and subsequently used during controlling activities (e.g.,controlling sounds, displays, haptic feedback and/or other actions onthe handheld device itself; or controlling virtual objects and/oractions displayed on one or more connected devices). In addition, thetiming of a tap, particularly related to the timing of one or moreprevious taps and/or other events in the environment of the device user(including taps on one or more other handheld devices) may be componentsof additional control functions by handheld device users.

In accordance with an example, a handheld device is provided forinteraction by a device user that includes a device body configured tobe held by a first hand of the device user; electronic circuitry withinthe device body that includes a device processor; at least one inertialmeasurement unit within the device body operatively coupled to thedevice processor; and at least one device display affixed to the devicebody operatively coupled to the device processor, wherein the deviceprocessor is configured to: generate a first illumination pattern by theat least one device display; acquire inertial measurement data from theat least one inertial measurement unit; determine, based at least inpart on the inertial measurement data, an initial tap location by thedevice user on one of the device body and the at least one devicedisplay; and generate a second illumination pattern by the at least onedevice display based at least in part on the initial tap location.

In accordance with another example, a handheld device is provided forinteraction by a device user that includes a device body configured tobe held by a first hand of the device user; electronic circuitry withinthe device body that includes a device processor; at least one inertialmeasurement unit within the device body operatively coupled to thedevice processor; and a speaker within the device body operativelycoupled to the device processor, wherein the device processor isconfigured to: generate a first sound by the speaker; acquire inertialmeasurement data from the at least one inertial measurement unit;determine, based at least in part on the inertial measurement data, aninitial tap location by the device user on the device body; and generatea second sound by the speaker based at least in part on the initial taplocation.

In accordance with yet another example, a handheld device is providedfor interaction by a device user that includes a device body configuredto be held by a first hand of the device user; electronic circuitrywithin the device body that includes a device processor; at least oneinertial measurement unit within the device body operatively coupled tothe device processor, wherein the device processor is configured to:acquire inertial measurement data from the at least one inertialmeasurement unit; compute directional data and magnitude data from theinertial measurement data; determine, based at least in part on one orboth of the directional data and the magnitude data, an initial taplocation by the device user on the device body; and perform an actionbased at least in part on the initial tap location on one or both of thedevice processor and a remotely connected processor.

Other aspects and features including the need for and use of the presentinvention will become apparent from consideration of the followingdescription taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding may be derived by referring to theDetailed Description when considered in connection with the followingillustrative figures. In the figures, like-reference numbers refer tolike-elements or acts throughout the figures. Presented examples areillustrated in the accompanying drawings, in which:

FIG. 1A illustrates the use of an index finger of a hand that opposesthe hand holding a handheld device to tap on the left-most devicedisplay (projecting the letter “C”) as a means of interaction by adevice user.

FIG. 1B follows on from the scenario shown in FIG. 1A, in which theindex finger of the hand that opposes the hand used to cradle thehandheld device taps on the center device display (projecting the letter“A”) as a means of machine-based interaction.

FIG. 2 illustrates the use of a thumb of the same hand used to hold thehandheld device to, as a means for interaction, tap on the upper body ofthe handheld device including in a region containing a device speaker(e.g., in response to interactive sounds produced by the speaker).

FIG. 3A illustrates tapping the center display (projecting images ofaquatic animals) of a handheld device against a rigid object (e.g.,surface of a table or desk) as a means of interaction by a device user.

FIG. 3B, similar to the scenario shown in FIG. 3A, illustrates tappingthe rightmost display (shown projecting symbols that represent phoneticsounds) onto a rigid object (e.g., surface of a table or desk) as ameans for human-machine interaction.

FIG. 4A shows exemplary sampled data from three IMU accelerometerchannels and a computed acceleration magnitude trace during a time whenthe left display of a handheld device was tapped using a finger (seeFIG. 1A), illustrating methods that may be used to detect the occurrenceand location of a tap.

FIG. 4B shows exemplary acceleration data and computed accelerationmagnitude traces during a time when the right display of a handhelddevice was tapped by a finger, illustrating differences in accelerations(i.e., compared with FIG. 4A showing traces when the left display wastapped) that may be identified to help determine tap location.

FIG. 4C shows exemplary acceleration data and computed accelerationmagnitude traces during a time when the right display of a handhelddevice was struck against a desktop, illustrating differences in traces(i.e., that may be used to identify tap modes or attributes) when adisplay was struck against a solid surface versus tapping the rightdisplay with a finger (e.g., compared with traces shown in FIG. 4B).

FIG. 5 demonstrates examples of coordinate systems and rotational axesthat may be used to describe movements, velocities and accelerations indifferent dimensions as a component of processes to compute taplocations on the handheld device from acquired IMU data.

FIG. 6 is a flowchart outlining exemplary steps to process and locate atap-based interactions following projection of an illumination patternon one or more handheld device displays.

FIG. 7 is a flowchart outlining exemplary steps to process and locate atap-based interactions following a sound generated by a handheld devicespeaker or buzzer.

DETAILED DESCRIPTION

Before the examples are described, it is to be understood that theinvention is not limited to particular examples described herein, assuch may, of course, vary. It is also to be understood that theterminology used herein is for the purpose of describing particularexamples only, and is not intended to be limiting, since the scope ofthe present invention will be limited only by the appended claims.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art. It must be noted that as used herein and in the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise. Thus, for example,reference to “a compound” includes a plurality of such compounds andreference to “the polymer” includes reference to one or more polymersand equivalents thereof known to those skilled in the art, and so forth.

Where a range of values is provided, it is understood that eachintervening value, to the tenth of the unit of the lower limit unlessthe context clearly dictates otherwise, between the upper and lowerlimits of that range is also specifically disclosed. Each smaller rangebetween any stated value or intervening value in a stated range and anyother stated or intervening value in that stated range is encompassedwithin the invention. The upper and lower limits of these smaller rangesmay independently be included or excluded in the range, and each rangewhere either, neither or both limits are included in the smaller rangesis also encompassed within the invention, subject to any specificallyexcluded limit in the stated range. Where the stated range includes oneor both of the limits, ranges excluding either or both of those includedlimits are also included in the invention.

Certain ranges are presented herein with numerical values being precededby the term “about.” The term “about” is used herein to provide literalsupport for the exact number that it precedes, as well as a number thatis near to or approximately the number that the term precedes. Indetermining whether a number is near to or approximately a specificallyrecited number, the near or approximating unrecited number may be anumber which, in the context in which it is presented, provides thesubstantial equivalent of the specifically recited number.

As introduced in the Summary above, a tap may be generated byintentionally moving and subsequently causing an object (i.e., a“striking object”) to hit a location on the surface of a handheld device(i.e., “tap location”) selected and/or targeted by the user. Thestriking object may, for example, be a finger of an opposing hand, anyother body part, a digit of the hand holding the device, a stylus, astick, a pencil, and so on.

Tap forces may also be generated by causing (i.e., moving with intent)the target location or area on the surface of the handheld device tocollide with a solid surface (or, at least a partially rigid surfacesuch as human skin, stuffed toy or page within a book). Combining suchmotions, both the handheld device and another object (e.g., child's toy,hand) may be moved simultaneously toward each other. In general,relative movements of the handheld device compared with the surfacebeing tapped determine tap characteristics (e.g., peak force,accelerations, computed tap location) versus which object is moved(e.g., relative to the ground or other objects in the environment of auser). IMU data streams prior to and following the tap may help todetermine whether a striking object was used to tap a stationary device,the device was forcefully moved toward another object (e.g., see FIG.4C), or both processes occurred simultaneously.

As introduced in the Background section above, IMU data streams may becomprised of one or more of:

-   -   1. up to three channels (i.e., representing three spatial        dimensions often denoted X, Y and Z; see FIG. 5 ) of        accelerometer data, and    -   2. up to three channels (i.e., representing rotation around 3        axes; see FIG. 5 ) of gyroscope rotational velocities.        Both acceleration and gyroscopic data may be expressed as one or        more time-varying vectors using, for example, Cartesian, polar        and/or spherical coordinate systems (see FIG. 5 ).

Optionally, data streams indicating orientations of the handheld devicerelative to either or both of the gravitational and magnetic pull orattraction of the earth may be considered. Such orientation data may bea component of processes to determine tap location. For example, if adevice is being cradled within a hand “upside-down” (or, at least notbeing held in an intended manner) by a device user (i.e., in any of upto three spatial dimensions), then expected acceleration and/orgyroscopic data streams may be considered using transformed coordinatesystems that reflect a measured or atypical device orientation. Inaddition, potential enumerated tap locations may be redefined based onthe possibility that one or more surfaces of the handheld device may bemore accessible and others hidden as a result of such repositioning bythe hand holding the device.

Device orientation data may also be used to express movements of thehandheld device relative to previous movements of the device (e.g., whensensing sequential rotation of the device similar to turning atraditional knob), and/or motions relative to other objects in theenvironment of the device user such as a viewable display screen. Withinadditional examples of implementations, if the orientation (e.g.,relative to gravitational attraction) of a vertically oriented displayscreen is known and IMU data streams include the direction ofgravitational pull relative to the body of the handheld device, thendisplayed images of the movements of the handheld device (or any virtualobject being controlled by the device such as a puppet or virtual toy)may be made to appear in the same orientation as the physical device, orany other selected viewing perspective. Within further examples, theorientation of a device (i.e., relative to the magnetic or gravitationalattraction of the earth) during a tap may be used to define one or moretap attributes (described more fully below) that may be used as acomponent of controlling functions of the handheld device by controllingor modulating actions on the device itself and/or on a connected device(e.g., a connected tablet).

Determining a tap location on the handheld device may be considered aclassification process that uses patterns (e.g., compared with atemplate) or “signatures” within IMU data streams to compute mostprobable tap locations. An initial step in such classification processesis to identify one or more methods to specify potential tap locations onthe surface of the handheld device. One method to indicate potential taplocations involves coating the surface of the device with a virtual mesh(e.g., made up of triangular, finite elements) or rectangular gridpattern (i.e., within in a curved two-dimensional or three-dimensionalspace) where mesh or grid intersections may represent potential taplocations. This process effectively assigns the surface of the handhelddevice with a uniform spatial distribution of potential tap locations.One disadvantage of this approach is that taps at two or more nearbygrid locations might not produce (i.e., at least within the variabilityof different users, different tap strengths, different hand grips, andso on) tap signatures that can be distinguished (i.e., uniquelyclassified) from each other.

Another exemplary method to identify tap locations involves enumeratinga desired number of distinct locations on the device surface.Effectively, enumeration allows multi-dimensional locations on thedevice to be collapsed into a single target classification data set,where the number of potential tap locations is related to (i.e.,effectively defines) a spatial resolution for tap locations withinregions. Such enumeration schemes allow the density of potential taplocations to vary over the surface of the device to, for example,account for lower spatial resolution when determining tap location insome regions of the device (e.g., in regions normally covered by a hand)and/or when using applications that may benefit from more closely spacedtap locations (e.g., in regions around one or more displays, see FIG.1A).

As examples during different applications, only three tap locations(e.g., on each of three displays, see FIGS. 1A and 1B) may be requiredwhen responding to simple queries; whereas other applications mayutilize or attempt to discern dozens of different tap locationsthroughout the various surfaces of the device. In addition to taplocations on one or more affixed displays (e.g., see FIGS. 1A and 1B),tap locations may include a region of a speaker or buzzer (see FIG. 2 ),anywhere on a top surface of the device body, a device edge or corner,or any of the surfaces on the opposite sides or edges of the device bodyor exterior elements just listed.

Conversion of analog IMU data into a digital form, suitable fornumerical processing, may use analog-to-digital (A/D) conversiontechniques, well-known in the art. IMU sample rates may generally be ina range from about 100 samples/second to about 10,000 samples/secondwhere (as described further in the Background section, above) higher IMUsample rates involve trade-offs involving signal noise, cost, powerconsumption and/or circuit complexity. Because a tap is a relativelyrapid event (e.g., generally lasting in a range from about 10 to 100milliseconds), higher IMU A/D sample rates may allow more distinct taplocations on the handheld device to be discerned.

Within examples of implementations, IMU sampling rates may be changeddynamically. For example, during most times, slower sample rates may beused to conserve power and computing resources (e.g., when discerningnon-tap, hand movement-based gestures). As soon as an increased signal(i.e., indicative of a tap) is sensed within accelerometer and/orgyroscope data streams, A/D sample rates may be increased up to themaximum available for the IMU device. Alternatively, or in addition,high-rate sampling may be buffered (e.g., held within a circular buffer)via hardware or firmware. When a potential tap is detected, one or moreprocessors may retrieve samples within the buffer prior to the detectionof the tap to more accurately assess, for example, an earliest time of atap (e.g., extrapolating one or more signals exceeding a threshold backto baseline) as well as to include additional samples for classificationof acquired data into a tap location. Within further alternative oradditional examples, hardware may perform threshold detection (see FIGS.4A, 4B and 4C), interrupting execution within one or more processingelements (i.e., executing firmware or software) whenever a tap isdetected by hardware elements.

In broad terms, classification processes to convert IMU data into a taplocation may include two overall approaches: 1) numerical approachesthat may include multi-channel template matching and/or frequency domain(e.g., Fourier transform) analyses, and 2) neural network approaches inwhich a network is trained using data sets of taps at known locationscollected under a variety of conditions (e.g., different users, distinctdigits or other objects used to tap, various hand sizes and/or positionsholding the device, varying tap forces, different device orientations).Such training data sets may be acquired from an individual user (orsmall number of users) who (e.g., due to age or ability to articulate)may have limited ranges of force and/or tap locations. Alternatively, orin addition, a wide range of users may be used to generate neuralnetwork training data sets (e.g., particularly involving taps usingfingers) that may then be applied globally (i.e., to any device user).

Numerical approaches may utilize a computational model of the handhelddevice coupled with contact locations by the hand holding the device.From such considerations, expected magnitudes, timing and directions offorces applied during taps at different locations on the device may beestimated. Template matching techniques may then be applied on all, or asubset, of IMU data streams. For example, template waveforms (e.g.,modelled and/or empirically acquired) from each potential tap locationmay be compared with acquired data to compute correlation coefficients(or similar measures). Template waveforms associated with a locationwith the highest correlation corresponds to a most likely tap position.Alternatively, or in addition, simply comparing the sequence of positiveand negative peaks within acquired data to those within templatewaveforms (e.g., across all available IMU channels) may be sufficient touniquely identify a tap location, particularly when (e.g., based oninteractive activities being performed) there is a small number oftarget tap locations.

During such calculations, it may be helpful to use one or more polarand/or spherical coordinate systems to more readily considerthree-dimensional (e.g., rotational) velocity and acceleration vectors,for example, about contact points including the functional center of theIMU and/or the hand holding the device. It may also be helpful toconvert time-based IMU data streams into a frequency domain (e.g.,processed using multidimensional Fourier transforms) to determinepresence of characteristic frequency components and phase differences(i.e., related to time and/or sequence) among frequency componentswithin data streams that may help to distinguish different taplocations.

Neural network approaches may include using the IMU data streams asinputs to one or more neural networks comprising, for example, either asingle output that identifies a maximum likelihood tap location (e.g.,indicating an enumerated tap location) or a binary (i.e., yes/no) outputassociated with each location. Either neural network configuration mayadditionally be configured to supply a confidence level (or anequivalent measure) related to the degree of match at each potential taplocation. Neural networks may also be trained to output one or more tapattributes described in more detail below. Tap attributes include, forexample, forcefulness of the tap, handheld device orientation, devicemovements at the time of the tap, whether struck by a soft or rigidobject, whether struck by a finger of an opposing hand (versus the handholding the device), and so on.

Techniques known in the art for digital signal processing (DSP) of suchtime-varying data streams may be used where: 1) multi-layer perceptron(MLP) and 2) recurrent neural network (RNN) are examples of networktopologies (i.e., suitable for processing time-based data streams).Back-propagation may typically be used to train such networks.Optionally, neural networks may be made adaptive by noting whencorrections are made within virtual activities controlled by thehandheld device. Such adaptive corrections may use supervised learningapproaches to further train a neural network.

Depending on factors such as IMU sample rate, noise within IMU data,mechanical design of the handheld device, computational approaches andtemporal/mechanical consistency of taps produced by a user (or otherusers of the device), algorithms to convert IMU data into tap locationsmay target any number of distinct tap locations. Tap locations on thehandheld device may be used as an input to control or modulate resultant“actions” in real and/or virtual worlds. Such actions may be: 1)strictly confined to the handheld device such as a haptic vibration,sound or symbol displayed on the handheld device, 2) transmitted toanother device in the environment of the device user (e.g., soundgenerated by a nearby speaker or image displayed on a nearby screen),and/or 3) transmitted to remote devices (i.e., handheld or otherwise)where taps and/or their locations may signal displays, audio and/orother indicators to remotely connected individuals.

As described further below, the time of occurrence of a tap (includingintervals between taps) may additionally be used as a component tocontrol or modulate actions ensuing from the tap. A number of methodsmay be used to determine a time of occurrence of a tap including:

-   -   1. the time of an initial detection of an acceleration or        velocity component greater than a threshold level, optionally        including techniques involving extrapolation of multiple samples        collected prior to exceeding the threshold, back to a baseline        as a means to increase precision of the initiation time        estimate,    -   2. the time of maximum (i.e., peak) acceleration or velocity,        that also may consider multiple sample points (e.g., fit to a        parabola) to increase the precision of determining such peaks,        and/or    -   3. midway (or any other selected reference) between initially        exceeding a threshold level and returning to less than the        threshold, where extrapolation techniques (e.g., linear fitting)        that consider multiple samples within threshold regions may be        used to increase precision.

In addition to using the time between taps as one input to control ormodulate resultant actions, the time between a “stimulus” such as ahaptic vibration, sound or symbol displayed on the handheld device, orother stimulus sensed in the environment of the device user (e.g.,broadcast sound, image displayed on a distant screen) versus a responsetap time may optionally be used as an additional modulator of actions.Such tap-timing commands may be deliberate (i.e., where the device userintentionally times when one or more tap responses occur) or unforced(i.e., where the stimulus-response time is measured by the device in amanner that may be unknown to the user). Resultant actions controlled ormodulated, at least in part by stimulus-response tap timing may bewithin the handheld device (e.g., display of one or more symbols,generation of one or more sounds, haptic vibrations) and/or tap timesmay be transmitted to other devices (e.g., tablet, laptop, electronicbook) where resultant actions are performed or modulated, at least inpart by measurements of tap-timing produced on the handheld device.

Within additional exemplary configurations, times between taps may bemeasured using two or more handheld devices (e.g., by two or morecollocated individuals or who may be located remotely and connected viatelecommunications). For example, one device user may initiate an actionwithin a virtual environment by a tap on a first handheld device. Withina predetermined time, a second user may complete the virtual action bytapping on a second device. Measured times between taps and/or othercontrol functions using two or more handheld devices may be performed byany number of device users. Such multi-user, shared control using tapsmay be particularly useful during game-play activities and other sharedactivities within virtual environments. Shared control of suchactivities (including the use of handheld devices) is described in U.S.Pat. No. 11,334,178, filed Aug. 6, 2021 and application Ser. No.17/531,571, filed Nov. 19, 2021, the entire disclosures of which areexpressly incorporated herein by reference.

Optionally, within additional examples of implementations, deviceorientation may be used as an input to control or modulate resultantactions. Device orientation may be determined from IMU components thatsense the direction of the gravitational pull and/or magnetic field ofthe earth, as described in more detail above. As an example, a tapperformed when the handheld device is held substantially horizontallymay be used to indicate a “no” answer by the device user during adialog; whereas, when the handheld device is held up vertically, a “yes”answer may be indicated. Orientation of the handheld device mayadditionally be used over a continuous range of orientations during oneor more taps (e.g., partially up or down, analogous to the orientationsof hands on a clock). For example, handheld device orientation may beused to select a particular hue or color (i.e., over a continuous range)while drawing.

Along similar lines within further examples, the force of a tap (e.g.,determined particularly from the amplitudes of magnitude peaks withinIMU data streams (e.g., see FIGS. 4A, 4B and 4C) may be used to convey acontinuous range of control or modulation of resultant actions. Forexample, when an orientation of a device is used to indicate a “yes” or“no” answer, as just described, a forceful tap may be used to indicate astrong degree of certainty by a device user that a response is correct.Conversely, a light tap may be used to indicate an uncertainty in aresponse.

In addition to peak forces, other waveform characteristics or“signatures” within IMU data streams may be used to identify a source ofa tap that may, in turn, be used to control or modulate resultantactions. Such signatures with IMU data may include tap direction,duration, frequency components and phase differences that may, forexample, help identify which hand or a particular finger used togenerate a tap. At least within a small number of device users, tapcharacteristics may even distinguish which user is holding the devicebased on factors such as the size of a hand and how solidly the deviceis held (i.e., determining resultant device movements as a consequenceof the forces applied during a tap) and/or how the device is cradledwithin a larger versus smaller hand (i.e., resulting in differentcontact points that constrain device movements during a tap). In furtherexamples, tap signature may distinguish the use of a stylus or pencil(versus a finger) to tap a device.

In yet further examples, tapping the device onto a rigid, immovablesurface such as a desk or floor may be distinguished from a moreforgiving surface such as a page within a book or stuffed toy. Tappingonto a rigid structure, compared with tapping soft tissues of mostsuperficial body parts, generally produces tap forces (i.e., sensed byone or more IMUs) that are shorter in duration, higher in peakamplitude, and contain higher compression-wave frequency componentsthroughout the tap (see FIG. 4C). Conversely, tapping an object that mayprovide some “give” during the tap generally produces forces andresultant (i.e., measured) accelerations that are longer in duration,lesser in peak amplitudes and contain lower compression-wavefrequencies. The physical location of a handheld device as well as itsorientation in (three-dimensional) space may be further determined bycamera-based tracking of the handheld controller. Camera-based measuresof the handheld controller (e.g., orientation in the field-of-view ofthe camera including relative to other objects in the field-of-view,location, velocity, acceleration) may be combined with IMU-based datastreams to provide further control or modulation of resultant actions.Systems and methods to determine such camera-based measures aredescribed in U.S. Pat. No. 11,334,178, filed Aug. 6, 2021 andapplication Ser. No. 17/531,571, filed Nov. 19, 2021, the entiredisclosures of which are expressly incorporated herein by reference.

In summary, the identification of a tap and its location on the handhelddevice may be considered only one of a number of control features (e.g.,“tap attributes”) that may be used to direct or modulate (i.e.,influence) actions. The following list summarizes measured attributes ofhandheld device taps that, in turn, may facilitate interaction withactions and/or activities embedded within the handheld device, and/orthat may be transmitted to other electronic devices (e.g., tablets,laptops, electronic book or magazine). Tap attributes may be used tointeract with the real world (e.g., to control internet-of-thingscomponents such as light switches and thermostats) and/or within avirtual environment (e.g., to control cartoon-like avatars and virtualbook page turns):

1. The presence of a tap and its time of occurrence, particularlyrelated to the occurrence of other events in the environment of a deviceuser, may be used as a primary control feature. As a visual example, atap (i.e., anywhere on the handheld device) may be used to control thetime when to turn a page within a virtual book displayed on a connectedtablet device. As an auditory example, the presence of a tap may be usedto indicate a selected answer immediately following articulation of theanswer within a series of possible answers broadcast using a speaker(e.g., on the handheld device or a nearby electronic device).

2. The determined location of a tap on the handheld device may be usedto control or modulate actions, particularly when making a so-called“one-of-N” selection. As an example, the surface of the handheld devicemay be “mapped” to the appearance of a human body shape, avatar, animalor other object displayed on a connected screen. In this example, thetop, bottom and sides of the handheld device may correspond to the top,bottom and sides (i.e., a similar spatial alignment) of the objectappearing on the screen. Tapping at specific locations on the handhelddevice may be used to indicate that subsequent movements of the handhelddevice are reflected in motions of the component selected by the tap(e.g., head or arms of a puppet).

3. The applied force or strength used to tap may be measured (e.g.,within peak amplitudes of acceleration). Distinctions may be madebetween light taps and more forceful ones, for example, to indicateconfidence in an answer or urgency to enact an action under the controlof the handheld device.

4. In addition to the overall force applied during a tap, the (moresubtle) timing of different force characteristics may also be used toidentify or discern characteristics of the objects striking the handhelddevice. For example, the striking of a pencil or stylus may bedistinguished from the use of a (e.g., relatively softer) finger of anopposing hand. Slight movements prior to a strike may distinguish theuse of a finger on the opposing hand versus a digit of the hand holdingthe device. Considering device movement over an even longer period oftime may help distinguish whether an object is moved to strike thedevice versus moving the handheld device itself to stroke anothersurface.

The interval between any two taps may generally be precisely controlledby a device user and subsequently measured. As a simple example,distinguishing long-duration versus short taps provides a simple methodfor binary (e.g., “yes” versus “no”) control. The continuous nature ofmeasuring such intervals (i.e., at least up to an ability for a typicaluser to accurately indicate a temporal interval) may be particularlyuseful. As an example, an interval between taps may be used to controlscreen brightness (i.e., over a continuous range of screenillumination).

6. Determining the temporal alignment of multiple and/or patterns oftapping provides an even wider range of temporal control functions. Tapintervals may be compared to temporal standards (e.g., one-second,two-second, etc.) or to each other (e.g., short tap followed by twolonger taps) to encode meaning to tap sequences. Greatly extending thenotion of conveying control through tap patterns, taps may be used toindicate musical melodies, pre-assigned interpretations (e.g., longintervals to increment units of hours while short taps increment minuteswhile setting a clock), or even Morse code.

7. Movements of the handheld device between (as well as during) taps mayprovide further attributes during device-enabled control functions. Bothmagnitude and/or direction of movements (including velocity and/oracceleration) during tap and inter-tap intervals may be considered. As asimple example, a direction of movement before or following a tap may beused to indicate whether a page turn within a virtual book should be inthe forward or reverse direction. Within a further example that takesinto account both magnitude and directional of inter-tap measurements,if an object on a screen is a focus of attention (e.g., a personspeaking), a nearby displayed object may be specified by a handhelddevice user as a new focus by indicating distance and direction (i.e.,relative to the previous focal location) between two taps.

8. Orientation relative to either or both of the magnetic field andgravitational pull or attraction of the earth during a tap may provideadditional tap attributes. As an example of a binary selection, tappingthe handheld controller when raised vertically may be used to indicateapproval of a selection (e.g., indicated by another person, visualizedon a screen or vocalized via a speaker), whereas a tap when the deviceis held substantially horizontally may indicate disapproval. As afurther example involving a continuous range of control using deviceorientation, the handheld device may be held vertically, and thenrotated and tapped (e.g., while visualizing the face of a traditionalclock) to control the intensity of a connected (e.g., viainternet-of-things) light. If the orientation of one or more otherobjects (e.g., relative to the gravitational pull of the earth) isknown, then the orientation of the handheld device may be computedrelative to those objects. In other words, the gravitational pull of theearth may provide a common reference direction for both the handhelddevice and other objects. As an example, if a display screen is orientedvertically, then the three-dimensional orientation of the handhelddevice at the time of a tap may be used to compute a three-dimensionalviewing perspective of one or more items drawn on the screen (e.g.,including a puppet or an image of the handheld device itself).

9. Taps attributes generated on one handheld device may be combined(i.e., in real time) with taps attributes generated on one or moreadditional handheld devices. Additional devices may be collocated withthe device user and/or used by interconnected users located remotely. Asecond handheld device may even be controlled by the second hand of adevice user.

Considering these handheld device manipulations as an ensemble, a numberof tap attributes may be used in various combinations to intuitively andeffectively generate a large number of real-time controls within virtualenvironments that, in turn, may also control aspects of the real-worldenvironment. As an example of the latter, the controller may be used totune a multi-speaker sound system where, once a speaker-volume controlfunction has been selected, tap location on the handheld device is usedto designate which speaker or combination of speakers is to becontrolled. Multiple speakers may be specified by two or more rapid tapsat differing locations on the handheld device. The device may then berotated within a frontal plane to control the speaker(s) volume (e.g.,analogous to a volume control knob) along with movements in and out ofthis plane to control a balance of bass versus treble.

As a further example, in this case within a virtual environment, andusing simple, intuitive controls that may be suitable for a young child,the handheld device may be used to control a range of aspects of apuppet (or other character, avatar or object) displayed on one or moreconnected display devices (that may also include remotely connecteddisplay devices viewable by others). A single tap on the body of thehandheld device may designate that (rotational and/or translational)movements of the device may be reflected in overall body movements ofthe puppet. A single tap to a side of the handheld device may indicate adesire to control movement of a particular limb. Multiple, closely timedtaps may be used to indicate a desire to control multiple limbs at thesame time via movements of the handheld device.

As described in more detail above, the spatial or “location resolution”(i.e., number of distinct locations that may be determined by aclassification process) may vary within different applications, forexample, from just a few tap locations (e.g., to responds to a yes/noquestion) to a few dozen or more possible distinct tap locations (e.g.,to identify different anatomical locations and attire or actions to becontrolled on a puppet). The number of potential locations may even varyfrom tap to tap. For example, a response to a series of questions withina quiz may be selected from one of three possible answers (i.e., eachanswer indicated by a different tap location) to address an initialquestion, whereas the next question within the quiz may be selected fromsix possible answers (i.e., again, where each potential answer isassociated with a distinct tap location).

Within additional implementation examples, classifying dynamicallychanging numbers of locations may be addressed by using distinctclassification processes (e.g., distinct neural networks that have beentrained based on taps at different numbers of potential tap locations)or by combining the outputs of classification process to “lump together”two or more classified locations into a single result. The latterapproach may additionally lump together or combine confidence levels (orother statistical measures) to determine a single, most likelycumulative tap location or region. Such confidence levels may, forexample, be summed or accumulated to result in lumped or regionalconfidence levels. In some outlier cases when using this approach, thecumulative probability of a collection of two or more classifiedlocations with a modest degree of certainty may exceed a single locationwith a higher level of certainty but that is surrounded by classifiedlocations with low levels of certainty.

In further examples, the use of tap locations and/or signalling motions(e.g., following a tap) by a device used to control virtual actions orobjects may take into account interactive context, leading to a conceptof “interpretive control”. “Interpretive control” may relax constraintson tap location(s), orientation(s), and/or motion(s) specified by one ormore controllers based on context. For example, if responses to a visualor auditory query are intended to be based on tapping one of threedisplays and a correct response is indicated by tapping the left displayof the handheld device (e.g., see FIG. 1A), then interpretive controlmay allow a tap (e.g., particularly by a young child) located anywhereon the left side of the handheld device to be interpreted as a correctanswer.

Interpretive control may reduce the tap location accuracy, timingconstraints, repetition rates, and/or number of degrees of freedomrequired of controlling devices by making assumptions about user intentbased on interactive context. Examples of contextual interpretationsthat may lead to a relaxing of controller precision include specifying achoice from a limited number of viable selections, using one or moreprevious and/or frequent selections to arrange choices at tap locationsthat are more readily accessible (e.g., tapping on the displays asdepicted in FIGS. 1A and 1B), utilizing tap patterns that make similarselections easy to indicate (e.g., a double tap at a location indicatesthat a previous tap pattern at the selected location should berepeated), and so on.

Interpretive control may be applied not only to enact or modulatevirtual activities based on tap location, but also based on tapattributes such as device orientation and or movements during, orimmediately following, a tap. For example, device motion during and/orfollowing a tap may be used to hammer a virtual nail (e.g., shown on aconnected display device). Interpretive control may allow any devicemotion following a tap (e.g., in any direction) to result in videosequences of the nail being hammered (e.g., absent requirements todirectionally strike the nail head or any minimum movement velocity forthe nail to move within a virtual board). Further, using interpretivecontrol, the nail may be driven fully into a virtual board after only alimited number (e.g., three) of motions representing hammer strokes.Along similar lines, rotation of a handheld device following a tap maybe used to control a virtual screw driver where any degree of rotationmay result in a full turn of the screw and (similar to the hammering ofa nail, just described) a limited number of rotations may fully insertthe screw.

Interpretive control may be particularly useful within interactionsinvolving the very young, the elderly, or those with reduced motorand/or cognitive functions. Further aspects of “interpretive control”are more fully described in U.S. Pat. No. 11,334,178, filed Aug. 6,2021, and co-pending application Ser. No. 17/531,571 filed Nov. 19,2021, the entire disclosures of which are expressly incorporated hereinby reference. Determining context from audiovisual content andsubsequently generating interpretive control based on such contexts aremore fully described in U.S. Pat. No. 11,366,997, filed Apr. 17, 2021,the entire disclosure of which is expressly incorporated herein byreference.

Within further examples, “bimanual control” of one or more virtualobjects may be implemented by combining an ability to specify locationson a touch-sensitive display using one or more fingers (or employing oneor more pointing instruments, such as a stylus) of one hand withsubstantially simultaneously generating additional activity controland/or modulation functions via a handheld device using a second hand.Bimanual control that combines location determined on a touch-sensitivedisplay with measured features of a tap on a handheld device (e.g., taplocation, timing and/or attributes) may produce multiple degrees offreedom while controlling virtual objects or activities. In the case ofa single user, touching a touch-sensitive display with a finger of onehand may confine the ability to tap a handheld device to using a digitof the hand holding the device and/or to tapping the device against asolid surface. Aspects of bimanual control by a single device user aremore fully described in U.S. Pat. No. 11,334,178, filed Aug. 6, 2021,the entire disclosure of which is expressly incorporated herein byreference.

Bimanual control may also be implemented by specifying a location on atouch-sensitive screen by one user while, substantially simultaneously,a finger (or other tap mechanism) of a second user may generate taps ona handheld device. Such control functions by two separate users (who maybe interacting remotely via telecommunications) may cooperativelyperform and/or modulate virtual actions and activities. Bimanual,cooperative control implemented by two device users are furtherdescribed in co-pending application Ser. No. 17/531,571 filed Nov. 19,2021, the entire disclosure of which is expressly incorporated herein byreference.

Within additional examples, although not “handheld” in a strict sense,portable electronic devices may be affixed and/or manipulated by otherparts of the human body. A device in which the locations of taps aredetermined based on IMU data streams may, for example, be affixed to anarm, leg, foot, or head. Such positioning may be used to addressaccessibility issues for individuals with restricted upper limb and/orhand movement, individuals absent a hand, and/or during situations wherea hand may be required for other activities. Tap timing and locationsmay be tracked based on tap motions while held or generated by otherbody parts.

The handheld device may additionally include one or more of one or moretouch controls, one or more microphones, one or more scroll wheels, oneor more photodiodes, one or more cameras, an optical heart sensor, andan electrical heart sensor, each operatively coupled to the deviceprocessor. Coupled with tap location, timing and attributes, thesecomponents may provide additional means for user control of actionsusing the handheld device. In addition, a battery, providing power tothe electronic components, may allow the handheld device to operateuntethered from any other power source.

Within yet further examples, the sensing of tap location based on IMUdata streams may largely (although not necessarily) eliminate needs forpushbuttons and/or other forms of (e.g., finger-based) contact controls;however, the elimination of a physical switch does not imply that theappearance of a contact structure also be eliminated. Indeed, images ofpush buttons or any other symbols may be applied to (e.g., painted on)surfaces of the handheld device at different potential tap locations.For example, catering particularly to young children, different taplocations may be indicated by brightly colored circles (or other shapes)on the surface of the handheld device. Optionally, such symbols may beapplied using temporary adhesives (i.e., including as so-called“stickers) that may be exchanged for different user applications orsimply for fun as a child ages and/or develops different personalpreferences. Such adhesive and/or fastened components may be an elementof different skins and/or other accessories that may be used to“dress-up” a handheld device (e.g., as a doll, puppet or miniature toycar).

FIGS. 1A and 1B illustrate exemplary tap sequences that allow a deviceuser to indicate selections from available responses shown on displays12 a, 12 b, 12 c of a handheld device 10. Specifying such responses may,for example, be included within machine-based interactions duringeducational sessions designed to help a young child learn to spelland/or pronounce words. In this example, algorithms used to classify taplocations may be informed that possible outcomes are confined to thethree locations of device displays (i.e., a small number compared withtypical applications, potentially simplifying classification processes).

FIG. 1A shows the handheld device 10 being held in the right hand 14 aof a device user (except for hands, remainder of the body of the usernot shown). The handheld device includes three spherical components 11a, 11 b, 11 c that may be used as elements of camera-based tracking (notshown) of the device. Each of the three spherical components 11 a, 11 b,11 c includes a display 12 a, 12 b, 12 c that separately projects theletters “C” at 12 a, “A” at 12 b, and “T” at 12 c; together, making upthe word “CAT”. A grating that covers a speaker 13 within the handhelddevice is also visible from the viewing perspective of FIGS. 1A and 1B.

During a session using the electronic device, the young child may beasked to find the first letter, C, within the word “CAT”. Alternatively,or in addition, the child may be asked (e.g., via the speaker, 13) toindicate a letter that can produce a phonetic “k” sound. As illustratedin FIG. 1A, the child responds using the index finger 15 of a left hand14 b to strike or tap (i.e., via an up-and-down motion 16 a) the top ofthe sphere 11 a containing the display 12 a that projects the letter“C”. Classification of which sphere was tapped may then be performedbased on acquired IMU data to, in turn, determine if the answer selectedby the child was correct.

As illustrated in FIG. 1B, the child may then be asked to find the nextcharacter, “A”, and/or the sound associated with the phonetic symbol“ae”. A correct answer may be indicated by tapping anywhere on themiddle sphere 11 b of the handheld device 10. FIG. 1B shows the childusing the same up-and-down motion 16 b of an index finger 15 of a lefthand 14 b to strike the center sphere 11 b. Once struck, classificationprocesses to determine which sphere was tapped may be performed based onacquired IMU data that, in turn, allows a determination whether theanswer indicated by the tap was correct.

FIG. 2 illustrates an ability to indicate tap locations on the handhelddevice 20 using a single hand 24. In this case, the handheld device 20is shown cradled by the right hand of a device user. The right-hand palm(hidden by the device in FIG. 2 ) and fingers 25 b, 25 c, 25 d, 25 eexcluding the thumb 25 a are able to mechanically stabilize the overalldevice 20, allowing for movement of the right-hand thumb 25 a (i.e.,relative to the device 20) including reaching the device speaker 23 andthree spherical, camera-based tracking components 21 a, 21 b, 21 c,where each of the three spheres includes a display 22 a, 22 b, 22 c.

The area in the region of the speaker 23 and the three spheres 21 a, 21b, 21 c are readily accessible to the thumb 25 a. For most individuals(i.e., depending on manual dexterity) regions along the left side 27 aand on much of the top surface 27 b of the device 20 may also be tappedby curling the thumb 25 a. Individual fingers 25 b, 25 c, 25 d, 25 e maybe used to tap a side of the device 20. It is even possible to tap thebacksides of the displays 21 a, 21 b, 21 c, the right side of thehandset (not shown within the viewing perspective of FIG. 2 ) and/or thebottom (not shown) of the handheld device 20 by manually rotating thedevice (i.e., still using a single hand) prior to tapping.

In response to an auditory prompt using the device speaker 23 and/or avisual prompt projected by any of the displays 22 a, 22 b, 22 c, theuser may tap with a thumb 25 a to indicate a selected response. As shownin FIG. 2 , a simple up-and-down motion of the thumb produces a tap inthe region of the speaker 23. Alternatively, or in addition, a deviceuser may indicate a response and/or selection by tapping multiple times26 in approximately the same region of the device. Once struck,classification processes to determine which region of the device wastapped and/or the timing of multiple taps may be performed based onacquired IMU data.

FIGS. 3A and 3B illustrate moving the handheld device itself 30 tostrike or tap a selected region of the device against an object orsurface 37. A handheld device 30 is shown being held by the palm(blocked from view by the device) and fingers 35 b, 35 c, 35 d, 35 eincluding the thumb 35 a of a right hand 34 of the user. Each of thethree spherical elements 31 a, 31 b, 31 c affixed to the device containsa display 32 a, 32 b, 32 c. The leftmost display projects the word “dog”32 a. The center display 32 b shows images of animals and the leftmostdisplay 32 c illustrates symbols representing different phonetic sounds.A perforated cover over an embedded speaker within the device 30 canalso be seen within the viewing perspective of FIGS. 3A and 3B.

In FIG. 3A, in response to a visual prompt (e.g., on a display 32 b) orauditory cue (e.g., via the speaker 33), the device user may wish toindicate that the center display 32 b showing animals is a selectedresponse. The user may indicate this response by striking 36 a thebottom side of the center sphere 31 b against a solid surface such asthe edge of a table or desk 37. Once struck, classification processes todetermine which sphere was tapped (i.e., including identification that abottom display surface has been struck) may be performed based, at leastin part on acquired IMU data. Classification processes may alsodistinguish (e.g., based on large peak amplitudes and short duration ofaccelerations during the strike) that the device 30 was struck against arigid object versus a tap generated by a finger (i.e., generating lessermovement and/or acceleration forces by a softer contact surface).

Following on and as demonstrated in FIG. 3B, a desired response to asubsequent prompt may involve tapping the rightmost display 31 c (i.e.,displaying symbols representing phonetic sounds) against a solid surface37. In this case, the handheld device 30, including its three displays31 a, 31 b, 31 c and speaker 33 are rotated slightly by the right hand34 of the device user prior to performing an up-and-down tapping motion36 b against the corner of a table or desk 37. Once struck, based onacquired IMU data, classification processes may determine that the outeredge of the rightmost display sphere 31 c was struck onto a solidsurface 37 and that the device was moved (sideways) to produce the tap.

FIGS. 4A, 4B and 4C show examples of IMU accelerometer data acquiredwhile tapping a handheld device, illustrating some of the distinctivefeatures within data streams produced by different forms of taps, andtaps at different locations. In each plot, data from a three-axisaccelerometer are shown where the three orthogonal axes (denoted X, Yand Z) are oriented approximately along axes depicted in FIG. 5 .Accelerometer measurements for each axis were sampled at about 6,667samples/second. Treating accelerometer data as a three-dimensionalvector, the magnitudes of accelerations, |A|, are also shown, computedaccording to

|A|=√{square root over ((X _(i) −X _(b))²+(Y _(i) −Y _(b))²+(Z _(i) −Z_(b))²)}  (eqn. 1)

where xi, Y_(i) and Z_(i) represent accelerometer samples (i.e., where“i” represents sample index) in each of the three dimensions; and X_(b),Y_(b) and Z_(b) represent so-called “baseline” values in each of thesame three dimensions. Baseline values may take into account factorssuch as electronic offsets of samples and may be determined duringperiods (e.g., by computing average values to reduce the effects ofnoise) when there is no movement of the accelerometers.Three-dimensional acceleration directions may also be computed from suchdata streams; however, these are not shown in FIGS. 4A, 4B and 4C.Multi-dimensional IMU gyroscope data streams and pointing vectors towardthe gravitational and/or magnetic pull of the earth are also not shown.

FIG. 4A shows data acquired during a tap using an index finger of anopposing hand (i.e., relative to a hand holding the device) to tap aleft display of a handheld device, similar to the action represented inFIG. 1A. Acceleration traces in the X at 40 a, Y at 40 b and Z at 40 caxes were used to compute acceleration magnitude, |A| at 41, asdescribed by equation 1. In FIG. 4A, the vertical scales foracceleration were not calibrated in absolute terms; however, values in X40 a, Y 40 b and Z 40C dimensions may be compared relative to eachother. The time bar at 43 a represents 25 milliseconds.

The occurrence and time of occurrence 42 a of a tap was identified as aresult of the acceleration magnitude 41 exceeding a predeterminedthreshold 42 b. Such thresholds may take into account signal noise aswell as the amplitudes of accelerations that are the result of tapsversus device movements that are not taps (e.g., to perform other handgestures). As described above, the accuracy of determining the time ofthe tap may be increased by extrapolating samples in the region wherethe signal exceeds a threshold 42 a back to a baseline level. Amongother distinctive features within these accelerometer data (e.g.,compared with traces in FIGS. 4B and 4C), the initial acceleration of atap in the Y axis is displayed as a sharp rise in the positive direction42 d. As discussed further below, there is also a trace feature 42 c inthe Z axis about 5 milliseconds after initial tap contact that may bethe result of breaking contact between the finger and the left displayof the handheld device.

FIG. 4B shows data acquired during a tap using an index finger of anopposing hand to tap a right display of a handheld device. Accelerationtraces in the X at 44 a, Y at 44 b and Z at 44 c axes were used tocompute acceleration magnitude, |A| at 45, according to equation 1.Vertical scales for all acceleration traces 40 a, 40 b, 40 c, 44 a, 44b, 44 c are the same in FIGS. 4A and 4B. In FIG. 4B, the time bar 43 brepresents 25 milliseconds. The occurrence and time of occurrence 46 aof a tap was identified when the acceleration magnitude 45 exceeded apredetermined threshold 46 b.

In FIG. 4B (in contrast to FIG. 4A), the initial acceleration in the Yaxis 44 b following the tap at 46 a is in the negative Y direction 46 d.The initial positive acceleration 42 d resulting from tapping the leftdisplay shown in FIG. 4A versus the initial negative acceleration 46 dfollowing tapping of the right display in FIG. 4B may be a consequenceof the left and right displays being located on opposite sides of the Yaxis (see FIG. 5 ).

Similar to FIG. 4A, FIG. 4B also shows a trace feature 46 c in the Zaxis about 5 milliseconds after initial tap contact. Some tracescollected during other taps (not shown) reveal trace features indifferent axes at approximately the same times following a tap up tomagnitudes that approach the amplitude of the initial tap. This tracefeature may be a result of breaking contact between the finger and thedevice (e.g., that may exhibit mechanical variability from tap to tap inforces generated during separation).

FIG. 4C shows data acquired during a process of tapping the rightmostdisplay of the handheld device against a solid surface (i.e., a desktop)similar to the action represented in FIG. 3B. Acceleration traces in theX at 47 a, Y at 47 b and Z at 47 c axes were used to computeacceleration magnitude, |A| at 48, according to equation 1. Verticalscales for all acceleration data are about 60% greater (i.e., peakamplitudes are greater in FIG. 4C) compared with FIGS. 4A and 4B. InFIG. 4C, the time bar 43 c represents a duration of 25 milliseconds. Theoccurrence and time of occurrence 49 a of the tap was identified as aresult of the acceleration magnitude 48 exceeding a predeterminedthreshold 49 b.

In cases involving movement of the handheld device toward a solidsurface, the acceleration of the device itself may be viewed in datastreams (e.g., at 49 d). Peak amplitudes resulting from such devicemovements may generally be less than those produced by any form of tap.Additionally, peak amplitudes (e.g., at 49 e) of taps against a rigidsurface (e.g., desktop) may generally be greater than taps generatedusing a finger (or other, softer surface). Taps of the handheld deviceagainst a solid surface may also generate high-frequency accelerations(e.g., reverberations) for a period (e.g., about 6 milliseconds at 49 c)following the tap. As the handheld device is moved back after strikingthe solid surface, the magnitude of acceleration traces, |A| at 48, doesnot return quickly to baseline levels (e.g., at 490 until retractionmovements by the device user are fully completed. Such distinctions(e.g., FIG. 4C compared with FIGS. 4A and 4B) are examples of featuresthat may be used to distinguish taps that use a finger or other objectto strike the handheld device versus using the device to strike anotherobject.

FIG. 5 illustrates considerations for assigning coordinate systems thatmay be used, particularly during numerical approaches, to classify tapswhen determining tap location. The coordinate system shown in FIG. 5 iscomposed of traditionally labelled X at 55 a, Y at 55 b, and Z at 55 caxes with an origin 54 at the geometric center of the handheld device50. Additionally, rotational movements about the X at 56 a, Y at 56 band Z at 56 c axes (i.e., corresponding to pitch, roll and yaw) may beexpressed. Other coordinate systems are also possible including, forexample, using polar coordinates and/or placing the origin at thegeometric or operational center of an IMU component located internally(not shown) within the device 50. If the one or more IMU components havethe ability to sense gravitational and/or magnetic pull of the earth,then coordinate systems of the handheld device 50 may additionally beexpressed relative to a vector 57 b pointed in the direction of the pullof the earth 57 a.

Taps at different locations on the handheld device 50 produce differenttranslational and rotational movements (i.e., measured usingmulti-dimensional accelerometers and gyroscopes, respectively) that maybe expressed using such one or more coordinate systems. As illustratedin FIGS. 4A, 4B and 4C, IMU data streams may represent rotational and/ortranslational movements that result from taps at different (i.e., known)locations on the device 50 (i.e., a “forward” mathematical approach).Distinctions among such movements (i.e., the “reverse” mathematicalapproach) may then be used within algorithmic strategies (e.g.,numerical, neural network-based) to classify tap locations.

As an example, a tap at 52 a on the left sphere 51 a may distinctivelyresult in a rotational movement in the negative direction 56 b about theY axis 55 b, coupled with rotational movement in the positive direction56 a about the X axis 55 a. On the other hand, a tap at 52 b on thecenter sphere 51 b may produce little rotational movement 56 b about theY axis 55 b, but significant torque in the positive direction 56 a aboutthe X axis 55 a. In further contrast, a tap at 52 c on the right sphere51 c may distinctively result in a rotational movement in the positivedirection 56 b about the Y axis 55 b, coupled with rotational movementin the positive direction 56 a about the X axis 55 a. Within yet furtherdistinctions, a tap in the area of the speaker at 53 of the handhelddevice 50 may produce only minor rotational movement about any axis, butmeasurable translational movement in the negative Z axis 55 c direction.As one more example, a tap on the lower left side of the device at 58may primarily and distinctively result in significant rotationalmovement in the positive direction 56 c about the Z axis 55 c.

FIG. 6 is a flowchart that outlines exemplary steps to classify taplocations on a handheld device in response to images, symbols and/orillumination patterns generated on device displays. Initially 60 a, auser may grip the handheld device 61 a to enable tapping and viewingdisplays 62 a, 62 b, 62 c. Next 60 b, an initial illumination pattern(e.g., containing one 62a, two 62 b, or three dots 62 c) may beprojected by the displays 61 b, viewable to the device user. IMU dataare then acquired 60 c and analysed (e.g., including computed magnitudesof movements) to determine if movements exceed a threshold 61 c (i.e.,indicating the occurrence of a tap). If the threshold is not exceeded 60d, then further IMU data are gathered 61 d. If IMU data exceed thethreshold or other criteria used to determine the occurrence of a tap 60d, then data around the time of tap determination are gathered toperform tap location classification 60 e.

FIG. 6 illustrates the use of a neural network 61 e to perform theclassification process 60 e. In this exemplary approach, IMU data in theregion around the time of exceeding a threshold are input to a neuralnetwork. A most likely tap location (e.g., the leftmost sphericaldisplay 63) may be generated as a neural network output, optionally,along with other attributes such as a degree of confidence regarding themost likely tap location and measured forces applied during the tap(e.g., related to type of tap and/or other tap attributes).

In addition, temporal data about the tap are registered 60 f includingthe time of the tap 61 f and the interval since a previous tap 64, alongwith determining any patterns of activity involving two or more previoustaps. Based on tap location, along with other tap attributes such astiming and tap forces, a new display pattern may be generated 60 g forthe device display 61 g. The overall process may then be repeated 60 busing the new display pattern.

FIG. 7 is a flowchart that outlines exemplary steps to classify taplocations on a handheld device in response to auditory cues or soundspresented using a device speaker. Sounds may, for example, comprise oneor more words, one or more utterances, one or more phonetic sounds, oneor more musical sounds, one or more animal sounds, one or more naturesounds, one or more bell sounds, one or more chime sounds, and/or one ormore alerts.

Initially 70 a, a user may grip the handheld device 71 a in order toenable tapping and listening to auditory cues from the speaker 72. Next70 b, a sound pattern (in this case, a series of musical notes 71 b) isbroadcast on the speaker. IMU data are then acquired 70 c and analysed71 c to determine if movements exceed a threshold (i.e., indicating theoccurrence of a tap). If the threshold is not exceeded 70 d, then theprocess of gathering IMU data is repeated 71 d. If IMU data exceed athreshold or satisfy other criteria used to determine the occurrence ofa tap 70 d, then data around the time of tap determination are used toperform tap location classification 70 e.

FIG. 7 illustrates the use of a numerical approach 71 e to perform theclassification process 70 e. In this case, a tap on the lower left sideof the handset 73 by the index finger of the hand opposing the hand usedto grip the device may, for example, be determined based on a dominantrotational signal in the X-Z plane 71 e (i.e., counter clockwise whenviewed facing the upper portion of the device). A most likely taplocation is computed along with, optionally, other attributes such asdegree of confidence regarding the most likely tap location and forcesapplied during and following the tap (e.g., tap attributes).

Additional temporal data about the tap may be registered 70 f includingthe time of the tap 71 f and the interval since a previous tap 74, alongwith determining any patterns of activity when combined with previoustaps. Based on the newly determined tap location, along with other tapattributes, a new sound pattern 71 g may be generated for broadcast 70g. The overall process may then be repeated 70 b using the new auditorystimulus.

The foregoing disclosure of the examples has been presented for purposesof illustration and description. It is not intended to be exhaustive orto limit the invention to the precise forms disclosed. Many variationsand modifications of the examples described herein will be apparent toone of ordinary skill in the art in light of the above disclosure. Itwill be appreciated that the various components and features describedwith the particular examples may be added, deleted, and/or substitutedwith the other examples, depending upon the intended use of theexamples.

Further, in describing representative examples, the specification mayhave presented the method and/or process as a particular sequence ofsteps. However, to the extent that the method or process does not relyon the particular order of steps set forth herein, the method or processshould not be limited to the particular sequence of steps described. Asone of ordinary skill in the art would appreciate, other sequences ofsteps may be possible. Therefore, the particular order of the steps setforth in the specification should not be construed as limitations on theclaims.

While the invention is susceptible to various modifications, andalternative forms, specific examples thereof have been shown in thedrawings and are herein described in detail. It should be understoodthat the invention is not to be limited to the particular forms ormethods disclosed, but to the contrary, the invention is to cover allmodifications, equivalents and alternatives falling within the scope ofthe appended claims.

1. A method for a human to interact using a handheld device including a device processor, at least one device display affixed to the device body operatively coupled to the device processor, and at least one inertial measurement unit within a device body operatively coupled to the device processor, the method comprising: generating, on the at least one device display, an illumination pattern; acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit when the human generates a tap by tapping the device in response to the illumination pattern; computing, by the device processor, directional data and magnitude data from the inertial measurement data; determining, by the device processor, based solely on one or both of the directional data and the magnitude data, a tap location on the device body when the human generates the tap; and performing an action by one or both of the device processor and a remotely connected processor, based at least in part on the tap location.
 2. The method of claim 1, wherein the illumination pattern comprises one or more of one or more alphanumeric characters, one or more symbols, and an illumination source.
 3. The method of claim 1, wherein the tap is generated by the human by one of tapping a second hand onto the handheld device being held by a first hand, tapping a second hand digit onto the handheld device, tapping the second hand digit onto the at least one device display, tapping a first hand digit onto the handheld device, tapping a body part onto the handheld device, tapping the handheld device onto a solid object, tapping the at least one device display onto the solid object, tapping the handheld device against an additional handheld device, and tapping the at least one device display onto an additional handheld device display.
 4. The method of claim 1, wherein the device processor is further configured to determine, from the inertial measurement data, an orientation of the handheld device relative to one or both of an earth gravitational attraction and an earth magnetic attraction, and wherein the action is further based on the orientation of the handheld device.
 5. The method of claim 1, wherein the handheld device further includes a device haptic unit operatively coupled to the device processor, the method further comprising activating the device haptic unit to alert the human that the illumination pattern is being generated on the at least one device display.
 6. The method of claim 1, wherein the handheld device further includes a device speaker operatively coupled to the device processor, the method further comprising broadcasting one or more sounds by the device speaker to alert the human that the illumination pattern is being generated on the at least one device display.
 7. A method for a human to interact using a handheld device including a device processor, a device speaker within the device body operatively coupled to the device processor, and at least one inertial measurement unit within a device body operatively coupled to the device processor, the method comprising: generating, on the device speaker, one or more sounds; acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit when the human generates a tap by tapping the device in response to the one or more sounds; computing, by the device processor, directional data and magnitude data from the inertial measurement data; determining, by the device processor, based solely on one or both of the directional data and the magnitude data, a tap location on the device body when the human generates the tap; and performing an action by one or both of the device processor and a remotely connected processor, based at least in part on the tap location.
 8. The method of claim 7, wherein the one or more sounds comprise one or more of one or more words, one or more utterances, one or more phonetic sounds, one or more musical sounds, one or more animal sounds, one or more nature sounds, one or more bell sounds, one or more chime sounds, and one or more alerts.
 9. The method of claim 7, wherein the tap is generated by the human by one of tapping a second hand onto the handheld device being held by a first hand, tapping a second hand digit onto the handheld device, tapping the second hand digit onto the at least one device display, tapping a first hand digit onto the handheld device, tapping a body part onto the handheld device, tapping the handheld device onto a solid object, tapping at least one device display affixed to the device body onto the solid object, tapping the handheld device against an additional handheld device, and tapping the at least one device display onto an additional handheld device display.
 10. The method of claim 7, wherein the handheld device further includes a device haptic unit operatively coupled to the device processor, the method further comprising activating the device haptic unit to alert the human that the one or more sounds are being generated on the device speaker.
 11. A method for a human to interact using a handheld device including a device processor, and at least one inertial measurement unit within a device body operatively coupled to the device processor, the method comprising: acquiring, by the device processor, inertial measurement data from the at least one inertial measurement unit when the human generates an initial tap by tapping the device; computing, by the device processor, directional data and magnitude data from the inertial measurement data; determining, by the device processor, based solely on one or both of the directional data and the magnitude data, an initial tap location on the device body when the human generates the initial tap; and performing an action by one or both of the device processor and a remotely connected processor, based at least in part on the initial tap location.
 12. The method of claim 11, wherein the at least one inertial measurement unit comprises one or more of one or more accelerometers, one or more magnetometers and one or more gyroscopes.
 13. The method of claim 11, wherein the initial tap is generated by the human by one of tapping a second hand onto the handheld device being held by a first hand, tapping a second hand digit onto the handheld device, tapping the second hand digit onto the at least one device display, tapping a first hand digit onto the handheld device, tapping a body part onto the handheld device, tapping the handheld device onto a solid object, tapping at least one device display affixed to the device body onto the solid object, tapping the handheld device against an additional handheld device, and tapping the at least one device display onto an additional handheld device display.
 14. The method of claim 11, further comprising determining, by the device processor, one or more additional taps at one or more additional tap locations on the device body based on additional inertial measurement data acquired from the at least one inertial measurement unit, and wherein the action is further based on the one or more additional tap locations.
 15. The method of claim 14, further comprising determining, by the device processor, one or more inter-tap intervals between the initial tap and the one or more additional taps, and wherein the action is further based on one or more of an initial time of occurrence of the initial tap and the one or more inter-tap intervals.
 16. The method of claim 11, further comprising determining from the inertial measurement data, by the device processor, an orientation of the handheld device relative to one or both of an earth gravitational attraction and an earth magnetic attraction, and wherein the action is further based on the orientation of the handheld device.
 17. The method of claim 11, wherein the handheld device further includes a device haptic unit operatively coupled to the device processor, the method further comprising activating the device haptic unit to alert the human that the inertial measurement data are about to be acquired.
 18. The method of claim 11, wherein the handheld device further includes one or both of a Wi-Fi communications module operatively coupled to the device processor and a Bluetooth communications module operatively coupled to the device processor, the communications module communicating between the device processor and the remotely connected processor.
 19. The method of claim 18, wherein, when generating the initial tap, the action is additionally based on a screen location pointed to by the human on a touch-sensitive screen operatively coupled to the remote processor.
 20. The method of claim 11, wherein the handheld device additionally includes one or more of one or more push buttons, one or more touch controls, one or more microphones, one or more scroll wheels, one or more photodiodes, an optical heart sensor, and an electrical heart sensor; each operatively coupled to the device processor. 